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Benefit  and  Value  of  Measurement 


The  benefit  and  value  of 
measurement  comes  from  the 
decisions  and  actions  taken  in 
response  to  analysis  of  the 
data,  not  from  the  collection  of 
the  data. 
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Measurement  and  Analysis  in  Action 

_ & 


Corrective  actions 
to  improve  process 


Decision-making 


.'rV,' 


f\  Data  collection 


e 


A 


©Data  analyzed, 

interpreted,  &  stored 


Data  &  interpretations 
reported 


Data  stored 


A 


Data 

Repository 

Measurements  collected 


n. 

t 


Measurement  I 
Report  I 


mm 


Software  Engineering  Institute 


Carnegie  Mellon 


David  Zubrow,  October  2008 

©2008  Carnegie  Mellon  University 


4 
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Polling  Question 


To  what  extent  does  your  organization  take  steps  to  ensure  it  is 
getting  value  from  its  project  data? 

•  Not  at  all 

•  Somewhat 

•  A  great  deal 
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Outline 


The  Need  for  a  Measurement  and  Analysis  Infrastructure 
Diagnostic  (MAID) 

•  Measurement  errors  and  their  impact 
MAID  Methods 

•  Process  Diagnosis 

•  Data  and  Information  Product  Quality  Evaluation 

•  Stakeholder  Feedback 
Summary  and  Conclusion 
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Where  do  Measurement  Errors  come  Fromi 


Data  Entry  Errors 

•  Manual  data  entry 

•  Lack  of  integrity  checks 

Differing  Operational  Definitions 

•  Project  duration,  defect  severity  or  type,  LOG  definition,  milestone 
completion 

Not  a  priority  for  those  generating  or  collecting  data 

•  Complete  the  effort  time  sheet  at  the  end  of  the  month 

•  Inaccurate  measurement  at  the  source 

Double  Duty 

•  Effort  data  collection  is  for  Accounting  not  Project  Management. 

—  Overtime  is  not  tracked 
—  Effort  is  tracked  only  to  highest  level  of  WBS 
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Where  do  Measurement  Errors  come  From2 


Dysfunctional  Incentives 

•  Rewards  for  high  productivity  measured  as  LoC/Hr 

•  Dilbert-esque  scenarios 

Failure  to  provide  resources  and  training 

•  Assume  data  collectors  all  understand  goals  and  purpose 

•  Arduous  manual  tasks  instead  of  automation 

Lack  of  priority  or  interest 

•  No  visible  use  or  consequences  associated  with  poor  data  collection  or 
measurement 

•  No  sustained  management  sponsorship 

Missing  data  is  reported  as  a  valid  value 

•  Can’t  distinguish  0  from  missing  when  performing  calculations 


-  Software  Engineering  Institute 


Carnegie  Mellon 


David  Zubrow,  October  2008 

©2008  Carnegie  Mellon  University 


9 


Percent 


Purpose  for  Measuring  is  Understood 


Program  Executive  Project  Other 

Manager  Manager 


Source:  CMU/SEI-2006-TR-009 


Analyst  Programmer 

Occasionally 
■  Frequently 


Engineer 
1847  Responses 


Software  Engineering  Institute 


Carnegie  Mellon 


David  Zubrow,  October  2008 

©2008  Carnegie  Meiion  University 


10 


Are  Documented  Processes  Used? 


Fr6C|U6ncy  1 852  Responses 

Source:  CMU/SEI-2006-TR-009 
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What  is  Measurement  Error? 
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Single  Value:  Deviation  from  the  “true”  value 

•  Distance  is  1  mile,  but  your  odometer  measures  it  as  1 .1  miles 

•  Effort  really  expended  on  a  task  is  2.75  hours,  but  it  is  recorded  as  3 

Data  Set:  Error  introduced  as  a  result  of  the  measurement  process 
used 


•  Not  as  defined,  but  as  practiced 
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Gold  Standard:  Accuracy  and  Precision 
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Cost  of  Poor  Data  Quality  to  an  Enterprise  - 
Typical  Issues  and  Impacts 


Typical  Issues 

•  Inaccurate  data  [1  -5%  of  data  fields  are  erred] 

•  Inconsistencies  across  databases 

•  Unavailable  data  necessary  for  certain  operations  or  decisions 


Typical  Impacts 


Operational 


•  Lowered  customer  satisfaction 

•  Increased  cost 

•  Lowered  employee  satisfaction 


Tactical 


•  Poorer  decision  making  & 
decisions  take  longer 

•  More  difficult  to  implement 
data  warehouses 

•  More  difficult  to  engineer 

•  Increased  organizational 
mistrust 


Strategic 


•  More  difficult  to  set  strategy 

•  More  difficult  to  execute  strategy 

•  Contribute  to  issues  of  data 
ownership 

•  Compromise  ability  to  align 
organization 

•  Divert  management  attention 


Source:  Redman,  1998 
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Impacts  of  Poor  Data  Quality 


Inability  to 

•  manage  the  quality  and  performance  of  software 
or  application  development 

•  Estimate  and  plan  realistically 

Ineffective 

•  process  change  instead  of  process  improvement 

•  and  inefficient  testing  causing  issues  with  time  to 
market,  field  quality  and  development  costs 

Products  that  are  painful  and  costly  to  use  within  real- 
life  usage  profiles 


Bad  Information  leading  to  Bad  Decisions 
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Why  a  Measurement  and  Analysis  Infrastructure 
Diagnostic 


Quality  of  data  is  important 

•  Basis  for  decision  making  and  action 

•  Erroneous  data  can  be  dangerous  or  harmful 

•  Need  to  return  value  for  expense 

Cannot  go  back  and  correct  data  once  it  is  collected  - 
opportunity/information  lost 

Need  to  get  the  quality  information  to  decision  makers  in  an 
appropriate  form  at  the  right  time 

Keep  from  collecting  the  wrong  type  of  data 
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Polling  Question 


To  what  extent  does  your  organization  take  steps  to  ensure  the  quality 
of  its  project  data? 

•  Not  at  all 

•  Somewhat 

•  A  great  deal 
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Outline 


The  Need  for  a  Measurement  and  Analysis  Infrastructure 
Diagnostic  (MAID) 

•  Measurement  errors  and  their  impact 

MAID  Methods 

•  Process  Diagnosis 

•  Data  and  Information  Product  Quality  Evaluation 

•  Stakeholder  Feedback 

Summary  and  Conclusion 
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MAID  Objectives 


Compare  an  organization’s  current  measurement  and  analysis 
activities  against  a  defined  set  of  criteria 

•  Are  we  doing  the  right  things  in  terms  of  measurement  and  analysis? 

•  How  well  are  we  doing  those  things? 

•  How  good  is  our  data? 

•  How  good  is  the  information  we  generate? 

•  Are  we  providing  value  to  the  organization  and  stakeholders? 

Make  recommendations  for  improvement 

•  How  can  identified  gaps  or  weaknesses  be  addressed? 

•  How  can  we  prepare  for  achieving  higher  maturity? 

—  Many  mistakes  made  in  establishing  M&A  at  ML2  and  3  that  do  not 
create  a  good  foundation  for  ML4  and  5 
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Methods  Overview 


The  MAID  approach  includes 

•  a  thorough  review  of  measurement-based  planning  documents, 
processes/procedures,  analysis  results,  and  management  reports 

•  a  series  of  individual  and  group  interviews  with  personnel  who 

—  collect  measurement  data 

—  analyze,  interpret  and  report  the  measurement  information 

—  use  the  reported  data  to  make  decisions 

•  a  briefing  and  detailed  report  describing  the 
strengths  and  weaknesses  of  the  measurement 
program 
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Criteria  for  Evaiuation:  Measurement  Pianning  Criteria^ 


Measurement  Objectives  and  Alignment 

•  business  and  project  objectives 

•  prioritized  information  needs  and  how  they  link  to  the  business, 
organizational,  regulatory,  product  and/or  project  objectives 

•  necessary  organizational  and/or  software  process  changes  to 
implement  the  measurement  plan 

•  criteria  for  the  evaluation  of  the  measurement  process  and  quality 
assurance  activities 

•  schedule  and  responsibilities  for  the  implementation  of  measurement 
plan  including  pilots  and  organizational  unit  wide  implementation 


Adapted  from  ISO  15939. 
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Measurement  Planning  Criteria2 


Measurement  Process 

•  definition  of  the  measures  and  how  they  relate  to  the  information  needs 

•  responsibility  for  data  collection  and  sources  of  data 

•  schedule  for  data  collection  (e.g.,  at  the  end  of  each  inspection,  monthly) 

•  tools  and  procedures  for  data  collection 

•  data  storage 

•  requirements  for  data  validation  and  verification  procedures 

•  confidentiality  constraints  on  the  data  and  information  products,  and 
actions/precautions  necessary  to  ensure  confidentiality 

•  procedures  for  configuration  management  of  data,  measurement 
experience  base,  and  data  definitions 

•  data  analysis  plan  including  frequency  of  analysis  and  reporting 

Adapted  from  ISO  15939. 
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Criteria  for  Evaluation:  Measurement  Processes  and 
Procedures 


Measurement  Process  Evaluation 

•  Availability  and  accessibility  of  the  measurement  process  and 
related  procedures 

•  Defined  responsibility  for  performance 

•  Expected  outputs 

•  Interfaces  to  other  processes 

—  Data  collection  may  be  integrated  into  other  processes 

•  Are  resources  for  implementation  provided  and  appropriate 

•  Is  training  and  help  available? 

•  Is  the  plan  synchronized  with  the  project  plan  or  other 
organizational  plans? 
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Criteria  for  Evaluation:  Data  Definitions 


Data  Definitions  (meta  data) 

•  Completeness  of  definitions 

—  Lack  of  ambiguity 

—  Clear  definition  of  the  entity  and  attribute  to  be  measures 

—  Definition  of  the  context  under  which  the  data  are  to  be 
collected 

•  Understanding  of  definitions  among  practitioners  and  managers 

•  Validity  of  operationalized  measures  as  compared  to 
conceptualized  measure  (e.g.,  size  as  SLOC  vs.  FP) 
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Criteria  for  Evaiuation:  Data  Collection 


Data  collection 

•  Is  implementation  of  data  collection  consistent  with  definitions? 

•  Reliability  of  data  collection  (actual  behavior  of  collectors) 

•  Reliability  of  instrumentation  (manual/automated) 

•  Training  in  data  collection  methods 

•  Ease/cost  of  collecting  data 

•  Storage 

—  Raw  or  summarized 
—  Period  of  retention 
—  Ease  of  retrieval 
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Criteria  for  Evaluation:  Data 


Quality 

•  Data  integrity  and  consistency 

•  Amount  of  missing  data 

—  Performance  variables 
—  Contextual  variables 

•  Accuracy  and  validity  of  collected  data 

•  Timeliness  of  collected  data 

•  Precision  and  reliability  (repeatability  and  reproducibility)  of 
collected  data 

•  Are  values  traceable  to  their  source  (meta  data  collected) 

Audits  of  Collected  Data 
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Criteria  for  Evaluation:  Data  Anaiysis 


Data  analysis 

•  Data  used  for  analysis  vs.  data  collected  but  not  used 

•  Appropriateness  of  analytical  techniques  used 

—  For  data  type 
—  For  hypothesis  or  model 

•  Analyses  performed  vs.  reporting  requirements 

•  Data  checks  performed 

•  Assumptions  made  explicit 
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Appropriate  Analysis:  Types  of  Hypothesis  Tests 
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Analysis  Evaluation:  Appropriate  Modeling 
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Criteria  for  Evaluation:  Reporting 


Reporting 

•  Evidence  of  use  of  the  information 

•  Timing  of  reports  produced 

•  Validity  of  measures  and  indicators  used 

•  Coverage  of  information  needs 

-PerCMMI 
—  Per  Stakeholders 

•  Inclusion  of  definitions,  contextual  information,  assumptions  and 
interpretation  guidance 
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Criteria  for  Evaluation:  Stakehoider  Satisfaction 


Stakeholder  Satisfaction 

•  Survey  of  stakeholders  regarding  the  costs  and  benefits  realized  in 
relation  to  the  measurement  system 

•  What  could  be  improved 

—  Timeliness 
—  Efficiency 
—  Defect  containment 
—  Customer  satisfaction 
—  Process  compliance 


Adapted  from  ISO  15939. 
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Polling  Question 


Do  you  feel  your  organization  views  measurement  and  analysis  as  a 
process? 

•  Not  at  all 

•  Somewhat 

•  A  great  deal 
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Outline 


The  Need  for  a  Measurement  and  Analysis  Infrastructure 
Diagnostic  (MAID) 

•  Measurement  errors  and  their  impact 
MAID  Methods 

•  Process  Diagnosis 

•  Data  and  Information  Product  Quality  Evaluation 

•  Stakeholder  Feedback 
Summary  and  Conclusion 
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Summary 


Measurement  and  analysis  is  a  process 

•  It  needs  to  be  supported  to  be  institutionalized  and  effective 

•  Some  measurement  error  and  diminished  utility  will  result  from  choice 
of  measurement  infrastructure  elements,  procedures  and 
instrumentation 

Measurement  Infrastructure  Diagnostic: 

•  Characterizes  performance  of  measurement  system 

•  Identifies  improvement  opportunities  for: 

—  Measurement  processes  and  data  quality 

Good  information  from  high  quality  measures 
and  anaiyses  to  support  decision  making 
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3(n  #ob  Wc 
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#tI)ErS  Pring 
(doob  ®ata. 

[Attributed  to  W.  Edwards  Deming,  father  or  quality  revolution] 
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SEMA  Curriculum 


Implementing  Goal-Driven  Measurement 

•  Feb  24-26  in  DC,  June  9-10,  September  15-17,  December  1-3  in  DC 

Analyzing  Project  Management  Indicators 

•  March  10-11,  July  1 4-1 6,  October  6-8 

Improving  Process  Performance  using  Six  Sigma 

•  January  26-30,  April  20-24,  November  2-6 

Designing  Products  and  Processes  using  Six  Sigma 

•  May  18-22,  December  7-1 1  in  DC 
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Questions? 
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